Seqera platform set up

For the Seqera Platform set up (easier), just follow the batch-forge instructions from seqera

Manual set up

These instructions are mainly based on Luke Pembleton's blog and discussions with Eva Price.

0- Create a VPC, internet gateway and subnets as in Appendix A

1- Set up a new IAM user with appropriate permissions for running Nextflow pipelines on AWS

1.1- Create an IAM user (add programmatic access at later stage in step 5 below- not like in blogs). Don’t bother about creating the group and adding policies now, just create the user for now

1.2- Create a user group and add the previous user to it.

1.3- Go to user group and create inline polices as per the NextFlow website (https://www.nextflow.io/docs/latest/aws.html). I did this one by one.

Notes:

I did not include the ones described in the nextflow website for Fargate
Added batch:TagResource (as in my first attempt my jobs were failing with nf_user is not authorized to perform: batch:TagResource), and CloudWatch Logs:GetLogEvents, as someone also suggested to add this and I could see some warnings in my .nextflow.log files about it
For each service (Batch, EC2, S3…), specify “All” Resources, otherwise the policy will not save them (if I go back to check the policy, only the services that had “All” resources are listed)

After granting access to all buckets by attaching policy "s3:*" , a warning indicated I also needed to add other permissions, including IAM:passRole and others from S3 themselves, which were already added. I just ignored this.

Once all the permissions are added, click next and save the policy

1.4- Create a role to be able to use spot instances -> Roles -> Create Role -> AWS service, Use Case = EC2 -> AmazonEC2SpotFleetTaggingRole. Then click next until is created (didn’t need to add permission, just to add a name for the Role at the end)

1.5- Create access keys to give programmatic access to the user: User -> Security credentials -> Create access key -> Command Line Interface -> Create access keys

2- Create a custom nextflow AMI for AWS Batch

Amazon ECS is a container orchestration service that enables you to run and manage Docker containers on a cluster of EC2 instances. When you create an ECS cluster that uses EC2 instances, you need to specify an AMI for those instances. The AMIs are templates to create EC2 instances, as they contain the software needed to launch the instances (OS, apps, etc). These AMI can be custom AMIs or ECS-optimized AMIs, which are optimized for running containers on ECS. In summary:

ECS: Manages and orchestrates Docker containers.
EC2 Instances: Provide the compute resources for running containers when using ECS with EC2 launch type.
AMIs: Provide the template to create EC2 instances. ECS-optimized AMIs are tailored for running ECS workloads.

2.1- EC2 dashboard -> Instances -> Launch instance -> Browse more AMIs. Then Click AWS Marketplace AMIs -> search for ECS -> filter by “Free”. Select one of the recent ones -> click on Subscribe now

2.2- Select instance type = t2.micro, and In Key pair -> create new key pair (save the *.pem file)

2.3a- Check that in the network settings, “Auto-assign Public ID” is ENABLED, otherwise, click “Edit” and enable it (this happens if for example I used the default VPC from UCL-AWS, and not do step 0 – create a VPC)

2.3b- I do need to create a new security group that allows ssh traffic, otherwise I will not be able to ssh to the instance. Configure storage = 30Gb gp3 (free tier). Then click “Create Instance”

2.4- Check that the instance has aws-cli, otherwise you need to install it as described in Appendix B

2.5- Under the Instances menu select your relevant instance and click Actions -> Images and Templates -> Create Image. Take note of the AIM ID (not the name)

2.6- Once the AMI has been created, the instance can be stopped or terminated, as this instance was used solely for the purpose of creating the AMI

NOTE1: The point of this step is to get one of the existing AMIs, and to build up on that by adding the aws-cli to allow data movement to and from S3 buckets. But if the AMI I’m using (with Amazon Linux 2023) already has aws-cli installed, I guess there is no point of doing this? I am basically taking an image and making an image of the same image without having added anything to it.. :?

NOTE2: I'm also confused about the aws path given in the nextflow.config. If I actually give the actual aws path from the AMI, my jobs fail. I needed to add the default /home/ec2-user/miniconda/bin/aws even though my AMIs never used conda to install aws, either they came with it, or I did it as in APPENDIX B'

3- Create a batch environment

3.1- AWS Batch dashboard -> compute environments -> create -> EC2

3.2- Select “managed”. If this is the first time setting up a batch environment, AWS will create the relevant service role and instance role. Just ensure “Create new Role” is selected. Alternatively, select “AWSServiceRoleForBatch” service role, and “ecsInstanceRole” as Instance role -> Next

3.3- Click “Use EC2 Spot instances”. In Spot Flee Role, reload and choose the EC2SpotFleeTaggingRole that was created earlier in step 1.4. Leave 0 CPUs for minimum and desired, and whatever number of CPUs as max. Leave default “optimal” as Allowed Instance Types.

NOTE1: I originally was doing this on my personal account with the free tier, so I had this as min=0, max=4, and noticed that when I did my hello world tests, I had some m4.xlarge instances running! I changed this to min=0, max=1, but the next day had a Zero-spend budget exceeded email as that costed me £0.02. Be careful with this if you want to stay within the free tier.

NOTE2: if you want to stay within the free tier, and select MAX=1, then your jobs will stay in RUNNABLE forever. This is most likely because there are no instances wioth 1 CPU available via spot.

3.4- Under additional configuration, select the key pair created earlier in step 2.2. Choose “BEST FIT” as allocation strategy. Under “EC2 configuration” click Add EC2 configuration, select Amazon Linux 2023 as image type, and type the AMI ID created earlier in the “Image ID override” box -> Next

3.5- In Network configuration leave defaults as I’m not sure what to do here -> Create compute environment

3.6- In the other blog they add an additional step. I need to add this step because if I don’t do this, my jobs die with:

“Download failed: s3://lconde-nf-bucket/nf-wd/03/96ed2a8b6170edfb4d9708a330f9dc/.command.run to - An error occurred (403) when calling the HeadObject operation: Forbidden; upload failed: ./.command.log to s3://lconde-nf-bucket/nf-wd/03/96ed2a8b6170edfb4d9708a330f9dc/.command.log An error occurred (AccessDenied) when calling the PutObject operation: Access Denied”

Update the ecsInstanceRole

AWS Batch just created the ecsInstanceRole for us but we need to add the S3 role to allow access to data stored in S3 buckets. Go to the IAM service as done in Step 6 in the Setting up a programmatic user with IAM section. Under Roles enter ecs in the search box and click on the ecsInstanceRole. Click the Attach policies and enter S3 in the filter box and select the AmazonS3FullAccess, then click Attach policy.

You should have both the AmazonS3FullAccess and the AmazonEC2ContainerServiceforEC2Role policies attached to this role.

3.7- Still within Batch AWS dashboard -> Job queues -> Create. Choose Amazon EC2 -> name the queue -> connect to compute environment created earlier -> create job queue

4- Create a S3 bucket

4.1- Create a new private bucket in relevant region. I just added a name and left the rest with default values

4.2- Go to the bucket, and create a new folder within the bucket to serve as the Nextflow working directory (take note of the S3 URI address)

5- Set up nextflow config

process {
  cpus = 1
  memory = '6.GB'
  time = '1.h'
}
params {
  max_cpus = 1
  max_memory = '6.GB'
  max_time = '1.h'
}
profiles {
  local {
    process.executor = 'local'

    singularity.enabled = true
    singularity.autoMounts = true
  }
  myriad {
    process.executor = 'sge'
    process.penv = "smp"
    process.clusterOptions = { "-l mem=${task.memory.giga}G" }

    singularity.enabled = true
    singularity.autoMounts = true
  }
  aws {
    process.executor = 'awsbatch'
    process.queue = 'nf-queue'
    process.container = 'ubuntu'

    docker.enabled = true

    aws {
      batch {
        cliPath = '/home/ec2-user/miniconda/bin/aws'
//      region = 'eu-west-2'
//      jobQueue = 'nf-queue'
      }
//    client {
//      uploadMaxRetries = 5
//    }
      region = 'eu-west-2'
    }

    wave {
      enabled = true
//    containerConfig {
//       from = 'Dockerfile'
//    }
    }
    fusion {
      enabled = true
    }

    workDir = 's3://lconde-nf-bucket/nf-wd/'
    //params.outdir = 's3://lconde-nf-bucket/output'

  }
}

6- Run some tests

module load blic-modules nextflow
source ~/.bash_profile # to reset tower credentials

nextflow run lconde-ucl/hello_2 -r main --outdir results_hello -with-tower -profile local
nextflow run lconde-ucl/hello_2 -r main --outdir results_hello -with-tower -profile myriad
nextflow run lconde-ucl/hello_2 -r main --outdir results_hello -with-tower -profile aws
nextflow run lconde-ucl/hello_2 -r main --outdir s3://lconde-nf-bucket/results_hello -with-tower -profile aws

nextflow run nf-core/demo --input ./param_files/samplesheet_demo.csv --outdir results_demo -with-tower -profile local
nextflow run nf-core/demo --input ./param_files/samplesheet_demo.csv --outdir results_demo -with-tower -profile myriad
nextflow run nf-core/demo --input ./param_files/samplesheet_demo.csv --outdir results_demo -with-tower -profile aws

APPENDIX A.- Create a public subnet in my UCL-AWS account so that the instances and compute environments have internet access

Creating a public subnet in AWS involves a series of steps within the AWS Management Console. Here's a detailed guide on how to do it:

Prerequisites

You need an AWS account.
You should have a VPC (Virtual Private Cloud) already created. If you don't have one, you'll need to create it first.

Steps to Create a Public Subnet

Log into the AWS Management Console:
- Open your web browser and go to the AWS Management Console.
- Log in with your AWS credentials.
Navigate to the VPC Dashboard:
- In the AWS Management Console, in the services menu, type "VPC" and select "VPC" to open the VPC Dashboard.
Select Your VPC:
- In the left-hand menu, click on "Your VPCs."
- Select the VPC where you want to create the public subnet.
Create the Subnet:
- In the left-hand menu, click on "Subnets."
- Click on the "Create subnet" button.
- In the "Create subnet" page, do the following:
  - **Name tag:** Enter a name for your subnet (e.g., "Public Subnet 1").
  - VPC: Ensure the correct VPC is selected.
  - Availability Zone: Select an Availability Zone where you want the subnet to reside.
  - IPv4 CIDR block: Enter the IPv4 CIDR block for the subnet (e.g., 10.0.1.0/24).
Configure the Subnet:
- Click the "Create subnet" button to create the subnet.
Modify the Route Table:
- To make the subnet public, you need to associate it with a route table that has a route to the Internet Gateway.
- In the left-hand menu, click on "Route Tables."
- Select the route table associated with your VPC. If there is no suitable route table, you can create one by clicking the "Create route table" button.
- With the route table selected, click on the "Routes" tab.
- Click the "Edit routes" button, then "Add route."
  - Destination: Enter 0.0.0.0/0 (this is the route for all IPv4 traffic).
  - Target: Select your Internet Gateway (if you don't have one, you'll need to create it and attach it to your VPC).
Associate Route Table with Subnet:
- Click the "Subnet associations" tab.
- Click "Edit subnet associations."
- Select the public subnet you created and click "Save associations."
Enable Auto-assign Public IP:
- To automatically assign public IP addresses to instances launched in this subnet, go back to the "Subnets" section.
- Select your public subnet.
- Click on the "Actions" button, and then select "Modify auto-assign IP settings."
- Check the box "Auto-assign IPv4" and then click "Save."

APPENDIX B.- How to check if the AMI has aws-cli installed, and if not, how to install it

To SSH into your EC2 instance on AWS, you'll need the public IP address of your instance (find it in the AWS console -> instance) and the key pair file (.pem) associated with it with permissions set to allow only the owner to read and write (chmod 400 /path/to/your-key.pem). The username depends on the operating system of your EC2 instance. For Amazon Linux, the default username is ec2-user.

ssh -i nf_key_pair.pem [ec2-user@](mailto:ec2-user@18.175.122.220)18.130.78.47
sudo yum update
sudo dnf upgrade --releasever=2023.4.20240528

The version of amazon linux I have here already comes with aws.

In a previous instance I had created, aws was not available so I had to install it. I tried following the instructions from the blog but it did not work for me, as conda uses too much memory for my t2.micro instance. So installed aws-cli as described in https://docs.aws.amazon.com/cli/v1/userguide/install-linux.html. It needs python3.8+ that can be installed as described in https://stackoverflow.com/questions/61993914/how-do-i-install-and-use-python-3-8-on-an-amazon-ec2-linux-instance

sudo amazon-linux-extras install python3.8
sudo yum install -y unzip nano
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
aws –version

AWS set up

l.conde@ucl.ac.uk

17 July, 2024